Mobile terminal, image display device and user interface provision method using the same

ABSTRACT

A digital television includes: a display; a tuner; a wireless network interface to transmit data to a mobile terminal and receive audio information and screen; and a controller. The controller is configured to control a signal processor to decode the video signal and the screen information, control the display to display first image corresponding to the decoded video signal with second image corresponding to the decoded screen information, the second image corresponding to an image displayed on the mobile terminal, and receive a command from a remote controller. The controller is also configured to control the signal processor to process the audio signal and the audio information and to output the processed audio signal or the processed audio information to an audio output unit, and to control the signal processor to change the second image&#39;s size or position depending on the at least one command received from the remote controller.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of co-pending U.S. patent applicationSer. No. 16/777,231, filed on Jan. 30, 2020, which is a Continuation ofU.S. patent application Ser. No. 16/358,132 filed on Mar. 19, 2019 (nowU.S. Pat. No. 10,579,201 issued on Mar. 3, 2020), which is aContinuation of Ser. No. 15/896,924, filed on Feb. 14, 2018 (now U.S.Pat. No. 10,282,050 issued on May 7, 2019), which is a Continuation ofU.S. patent application Ser. No. 15/463,914, filed on Mar. 20, 2017 (nowU.S. Pat. No. 9,927,942 issued on Mar. 27, 2018), which is aContinuation of U.S. patent application Ser. No. 13/895,800, filed onMay 16, 2013 (now U.S. Pat. No. 9,632,648 issued on Apr. 25, 2017),which claims the benefit of Korean Patent Applications Nos.10-2012-0073677, filed on Jul. 6, 2012, and 10-2012-0082974, filed onJul. 30, 2012, all of which are hereby expressly incorporated into thepresent application.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to a mobile terminal, an image displaydevice and a user interface provision method using the same, and moreparticularly, to a mobile terminal, an image display device and a userinterface provision method using the same, which are capable ofdisplaying screens of various mobile devices on the image displaydevice, allowing a user to control the mobile terminals via the screensof the mobile terminals output on the image display device, outputtingthe screens of the mobile terminals, and controlling the screens outputon the image display device via the mobile terminals.

Discussion of the Related Art

Recently, with technological development, a mirroring service forsharing an image between two devices has been developed and has comeinto widespread use. The mirroring service is provided using a sourcedevice for providing image information and a sink device for outputtingthe received image information. Since the mirroring service convenientlyallows a user to share the screens of the two devices by displaying thescreen of the source device on the sink device, services associatedtherewith have been gradually increased.

Examples of the source device include a mobile device having arelatively small screen and configured to easily receive a user commandand examples of the sink device include an image display device having alarge screen.

Generally, the mirroring service is provided only using one sourcedevice and one sink device. That is, only the screen of one mobileterminal is displayed on one image display device. However, as userdemands for various screens have been increased, there is a need for amulti-mirroring service for connecting various source devices to onesink device and displaying the screens of the various source devices onone sink device.

A multi-mirroring service includes a connected sink device and a multiuser input back channel (UIBC) service provision method for controllinga sink device.

The related art mobile terminal is configured to receive a user commandusing a touch sensor. That is, the user touches a touch sensor of amobile terminal to enter a user command. The related art mobile terminaldoes not display a pointing cursor corresponding to the touch input.

Accordingly, if the user manipulates the screen of the image displaydevice which is the sink device via the mobile terminal which is thesource device, the user manipulates the mobile terminal and then checksthe image display device in order to check whether operation has beenproperly performed. That is, the user must perform the operation whilefrequently and alternately checking the mobile terminal and the imagedisplay device.

Accordingly, in the operation requiring considerable interaction betweenthe source device and the sink device, the user has to inconvenientlyperform operations while frequently and alternately checking the mobileterminal and the image display device. This is troublesome to a user whodesires to fix their eyes in one direction.

SUMMARY OF THE INVENTION

Accordingly, the present invention is directed to a mobile terminal, animage display device and a user interface provision method using thesame that substantially obviate one or more problems due to limitationsand disadvantages of the related art.

An object of the present invention is to provide a mobile terminal, animage display device and a user interface provision method using thesame, which are capable of allowing a user to control mobile terminalsvia the screens of the mobile terminals output on the image displaydevice and allowing the user to efficiently control the screens outputon the image display device.

Another object of the present invention is to provide an image displaydevice and a user interface using the same, which are capable ofallowing a user to efficiently control mobile terminals via the screensof mobile terminals output on the image display device.

Additional advantages, objects, and features of the invention will beset forth in part in the description which follows and in part willbecome apparent to those having ordinary skill in the art uponexamination of the following or may be learned from practice of theinvention. The objectives and other advantages of the invention may berealized and attained by the structure particularly pointed out in thewritten description and claims hereof as well as the appended drawings.

To achieve these objects and other advantages and in accordance with thepurpose of the invention, as embodied and broadly described herein, auser interface provision method using an image display device and atleast one mobile terminal connected to each other for data communicationincludes generating and transmitting screen information of the mobileterminal to the image display device; outputting a mirroring screenwhich is the transmitted screen information of the mobile terminal in afirst region of a display of the image display device; setting a secondregion corresponding to an adjacent region of the first region;outputting a cursor input through a user input unit of the image displaydevice on the display of the image display device; establishing datacommunication between a mobile terminal corresponding to the firstregion adjacent to the second region and the image display device if thecursor is located in the second region; determining that the mobileterminal corresponding to the first region is a target mobile terminalto be controlled by a user if the cursor is located in the first region;and transmitting a user command input through the user input unit of theimage display device to the target mobile terminal.

In another aspect of the present invention, an image display deviceconnected to at least one mobile terminal for data communicationincludes a display unit configured to output an image; a communicationunit configured to perform data communication with the mobile terminal;a user input unit configured to receive a user command from a user; anda controller configured to control the communication unit to receivescreen information of the mobile terminal, to control the display tooutput a mirroring screen which is the received screen information ofthe mobile terminal in a first region of a display of the image displaydevice, to control the display to set a second region corresponding toan adjacent region of the first region and to output a cursor inputthrough a user input unit of the image display device on the display ofthe image display device, to control the communication unit to establishdata communication with a mobile terminal corresponding to the firstregion adjacent to the second region and the image display device if thecursor is located in the second region, and to control the communicationunit to determine that the mobile terminal corresponding to the firstregion is a target mobile terminal to be controlled by a user if thecursor is located in the first region and to transmit a user commandinput through the user input unit of the image display device to thetarget mobile terminal.

In another aspect of the present invention, a mobile terminal includes adisplay configured to output an image; a communication unit configuredto perform data communication with an external image display device; asensor unit including a first touch sensor for sensing a first usertouch received from a user and a second touch sensor for sensing asecond user touch; and a controller configured to control the display tooutput a guide image in a screen region in which the first user touch isinput, to analyze and execute a user command corresponding to the seconduser touch, to generate screen information using an image output on thedisplay, and to control the communication unit to transmit the generatedscreen information to the image display device.

It is to be understood that both the foregoing general description andthe following detailed description of the present invention areexemplary and explanatory and are intended to provide furtherexplanation of the invention as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are included to provide a furtherunderstanding of the invention and are incorporated in and constitute apart of this application, illustrate embodiment(s) of the invention andtogether with the description serve to explain the principle of theinvention. In the drawings:

FIGS. 1A and 1B are diagrams showing a user interface provisionenvironment using a mobile terminal and an image display deviceaccording to an embodiment of the present invention;

FIG. 2 is a block diagram showing the internal configuration of an imagedisplay device according to an embodiment of the present invention;

FIG. 3 is a block diagram showing the internal configuration of a mobileterminal according to an embodiment of the present invention;

FIGS. 4 to 7 are diagrams illustrating a user interface provision methodaccording to a first embodiment of the present invention;

FIGS. 8 to 10 are diagrams illustrating a screen configuration method ofan image display device for providing a multi-mirroring serviceaccording to an embodiment of the present invention;

FIGS. 11 to 13 are diagrams illustrating a first embodiment of a multiUIBC service providing method of the present invention;

FIGS. 14 to 15 are diagrams illustrating a second embodiment of a multiUIBC service providing method of the present invention;

FIG. 16 is a flowchart illustrating a user interface provision methodaccording to a first embodiment of the present invention;

FIGS. 17 and 18 are flowcharts illustrating a user interface provisionmethod according to a second embodiment of the present invention;

FIGS. 19 to 23 are diagrams illustrating a first embodiment of a userinterface provision method for controlling the screen of an imagedisplay device of the present invention;

FIGS. 24 to 28 are diagrams illustrating a second embodiment of a userinterface provision method for controlling the screen of an imagedisplay device of the present invention;

FIGS. 29 to 33 are diagrams illustrating a third embodiment of a userinterface provision method for controlling the screen of an imagedisplay device of the present invention;

FIG. 34 is a diagram showing a user interface environment illustratingfourth to sixth embodiments of a user interface provision method forcontrolling the screen of an image display device of the presentinvention;

FIG. 35 is a diagram illustrating a fourth embodiment of a userinterface provision method for controlling the screen of an imagedisplay device of the present invention;

FIG. 36 is a diagram illustrating a fifth embodiment of a user interfaceprovision method for controlling the screen of an image display deviceof the present invention;

FIG. 37 is a diagram illustrating a sixth embodiment of a user interfaceprovision method for controlling the screen of an image display deviceof the present invention;

and

FIGS. 38A and 38B are diagrams illustrating a seventh embodiment of auser interface provision method for controlling the screen of an imagedisplay device of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

Reference will now be made in detail to the preferred embodiments of thepresent invention, examples of which are illustrated in the accompanyingdrawings. Wherever possible, the same reference numbers will be usedthroughout the drawings to refer to the same or like parts.

Although most terms of elements in this specification have been selectedfrom general ones widely used in the art taking into considerationfunctions thereof in this specification, the terms may be changeddepending on the intention or convention of those skilled in the art orthe introduction of new technology. Some terms have been arbitrarilyselected by the applicant and their meanings are explained in thefollowing description as needed. Thus, the terms used in thisspecification should be construed based on the overall content of thisspecification together with the actual meanings of the terms rather thantheir simple names or meanings.

The words “module” and “unit”, which are appended to terms describingcomponents, are merely used for ease of explanation of the presentinvention and thus may be used interchangeably.

An image display device 100 described in the present specification mayinclude all devices capable of displaying an image, such as a mobilephone, a smartphone, a personal digital assistants (PDA), a portablemultimedia player (PMP), a TV receiver, a laptop computer, a digitalbroadcast terminal, a navigation device, etc.

A mobile device 200 described in the present specification may includeall terminals connected to the image display device 100 to perform datacommunication, such as a mobile phone, a smartphone, a personal digitalassistants (PDA), a portable multimedia player (PMP), etc.

FIG. 1 is a diagram showing a user interface provision environment usinga mobile terminal 200 and an image display device 100 according to anembodiment of the present invention.

In the user interface provision environment of the present invention,the image display device 100 and the mobile terminal 200 are connectedby wire or wirelessly to perform data communication. That is, the imagedisplay device 100 and the mobile terminal 200 are configured totransmit/receive data to/from another device connected thereto by wireor wirelessly.

In a method of connecting the image display device 100 and the mobileterminal 200, various technologies such as Ethernet, universal serialbus (USB), mobile high-definition (MHL), Wireless LAN (WLAN) (Wi-Fi),Wireless broadband (Wibro), Bluetooth, radio frequency identification(RFID), Infrared Data Association (IrDA), Ultra-WideBand (UWB), ZigBee,etc.

The image display device 100 and the mobile terminal 200 may share ascreen through data communication. That is, the image display device 100and the mobile terminal 200 receive and output screen information of thecounterpart device. Thus, a user can receive the same screen using thetwo devices.

Sharing the screen between the two devices is referred to as a mirroringservice. The mirroring service may be provided using a source device forproviding screen information and a sink device for outputting thereceived screen information. That is, the mirroring service displays thescreen of the source device on the sink device.

The source device encodes the screen information of the source deviceand transmits the encoded screen information to the sink device. Thesink device decodes the received screen information and outputs thedecoded screen information. At this time, the source device may requesta decoding method supported by the sink device from the sink device andreceive the decoding method.

In addition, the source device may transmit audio information of thesource device to the sink device. The sink device outputs the receivedaudio information via a speaker. In this case, the source device mayseparately transmit screen information and audio information or maymultiplex screen information and audio information and transmit themultiplexed information.

In the present specification, the image display device 100 is the sinkdevice and the mobile device 200 is the source device. Alternatively,the image display device 100 may be used as the source device and themobile device 200 may be used as the sink device.

The image display device 100 and the mobile terminal 200 can transmit auser command input through data communication. That is, if the userenters a user command to the image display device 100 for displaying thescreen of the mobile terminal 200, the image display device 100transmits the user command to the mobile terminal 200 and the mobileterminal 200 performs operation of the mobile terminal 200 according tothe received user command.

Accordingly, the user can control the mobile terminal 200 using theimage display device 100. Controlling operation of the source deviceusing the sink device is referred to as a user input back channel (UIBC)service.

In FIG. 1A, the image display device 100 and the mobile terminal 200equally output an application icon screen which is the screen of themobile terminal 200 through data communication. That is, the mobileterminal 200 transmits the screen information of the mobile terminal 200to the image display device 100 through data communication and the imagedisplay device 100 displays the screen information of the mobileterminal 200.

The image display device 100 receives a user command from a user inputunit 155 such as a wireless remote controller and transmits the receiveduser command to the mobile terminal 200. That is, the user enters theuser command to the image display device 100 to control the mobileterminal 200.

In addition, as shown in FIG. 1A, the image display device 100 and themobile terminal 200 may display guide images 156 and 256 for displayingthe input of the user command on the respective screens.

FIG. 1A shows one mobile terminal 200 connected to the image displaydevice 100, and FIG. 1B shows two mobile terminals 201 and 202 connectedto the image display device 100.

Referring to FIG. 1B, connecting several mobile terminals 201 and 202 toone image display device 100 and simultaneously displaying the screensof the mobile terminals 201 and 202 on the image display device 100 isreferred to as a multi mirroring service. The user controlling severalmobile terminals 201 and 202 through the image display device 100 isreferred to as a multi UIBC service.

An object of a user interface of the present invention is to efficientlyprovide a multi mirroring service and a multi UIBC service to a user ifseveral mobile terminals 201 and 202 are connected to the image displaydevice 100.

In the present specification, prior to describing a user interfaceprovision method using the image display device 100 and the mobileterminal 200, the configuration of the image display device 100 and themobile terminal 200 will be described.

In particular, FIG. 2 is a block diagram showing the internalconfiguration of the image display device 100 according to an embodimentof the present invention.

Referring to FIG. 2, the image display device 100 includes anaudio/video processor 101, an interface 150, a memory 160, a display170, an audio output unit 175 and a controller 180.

Components shown in FIG. 2 are not essential and the image displaydevice 100 having more components or fewer components may beimplemented. Hereinafter, the above-described components will besequentially described.

The audio/video processor 101 processes a video or audio signal suchthat video or audio is output through the display 170 or the audiooutput unit 175 of the image display device 100. For video or audiosignal processing, the audio/video processor 101 may include a signalinput unit 110, a demodulator 120 and a signal processor 140. Inaddition, the signal input unit 110 may include a tuner 111, an A/Vinput unit 112, a USB input unit 113 and a wireless signal input unit114.

The tuner 111 tunes to a Radio Frequency (RF) broadcast signalcorresponding to a channel selected by a user from among RF broadcastsignals received through an antenna and converts the tuned RF broadcastinto an Intermediate Frequency (IF) signal or an analog basebandAudio/Video (AV) signal. The analog baseband audio/video signal (CVBS(Composite Video Banking Sync) sound intermediate frequency (SIF))output from the tuner 111 may be directly input to the signal processor140.

The demodulator 120 receives the digital IF signal DIF from the tuner110 and demodulates the digital IF signal DIF. The demodulator 120 mayperform demodulation and channel decoding, thereby obtaining a streamsignal TS. The stream signal output from the demodulator 120 may beinput to the signal processor 140.

The signal processor 140 performs demultiplexing and signal processing,outputs the video signal to the display 170 and outputs the audio signalto the audio output unit 175.

In addition, the signal input unit 110 may connect an external deviceand the image display device 100. The signal input unit 110 is connectedto an external device such as a digital versatile disc (DVD), a Blu-raydevice, a game console, a camcorder or a computer (laptop) to transmitan external input video signal, an external input audio signal and anexternal input data signal to the signal processor 140 of the imagedisplay device 100. The video signal, audio signal and data signalprocessed by the image display device 100 may be output to anotherexternal device.

The A/V input unit 112 of the signal input unit 110 may include aComposite Video Banking Sync (CVBS) port, a Component port, aSuper-video (S-video) (analog) port, a Digital Visual Interface (DVI)port, a High-Definition Multimedia Interface (HDMI) port, aRed-Green-Blue (RGB) port, a D-SUB port, an IEEE 1394 terminal, an SPDIFterminal, a liquid HD terminal, etc. in order to input the video andaudio signals of the external device to the image display apparatus 100.The analog signal input via the CVBS s-terminal and the S-video terminalmay be converted into a digital signal and the converted digital signalmay be input to the signal processor 140. The digital signal input viaanother input terminal may be input to the signal processor 140 withoutanalog/digital conversion.

The USB input unit 113 may receive the audio and video signal through aUSB terminal.

The wireless signal input unit 114 may connect the image display device100 to a wireless network. The image display device 100 may be connectedto the wireless Internet through the wireless signal input unit 114. Thewireless signal input unit 114 may perform short-range wirelesscommunication with other electronic devices.

The signal input unit 110 may connect the image display device 100 to aset-top box. For example, if the set-top box is an internet protocol(IP) TV set-top box, the audio, video or data signal of the IPTV set-topbox may be sent to the signal processor and signals processed by thesignal processor 140 may be sent to the IPTV set-top box, forinteractive communication. The IPTV may include an Internet TV and afull-browsing TV.

The signal processor 140 may demultiplex the received stream signal,e.g., MPEG-2 TS, into the video signal, the audio signal and the datasignal. The signal processor 140 may perform video processing of thedemultiplexed video signal. For example, if the demultiplexed videosignal is an encoded video signal, the signal processor 140 may decodethe encoded video signal. More specifically, if the demultiplexed videosignal is an MPEG-2 encoded video signal, the MPEG-2 encoded videosignal may be decoded by an MPEG-2 decoder.

The video signal processed by the signal processor 140 is displayedusing the display 170. The signal processor may perform audio processingof the demultiplexed audio signal.

The signal processor 140 may perform on screen display (OSD) processing.More specifically, the signal processor 140 may output a variety ofinformation on the screen of the display 170 in the form of graphic ortext based on at least one of the processed video signal and theprocessed data signal and the signal input through the mobile terminal200.

The memory 160 may store various applications necessary for thecontroller 180 to process and control signals, and may also storeprocessed video, audio and data signals. The memory 180 may temporarilystore a video, audio and/or data signal received from the signal inputunit 110.

The memory 160 may include at least one of a flash memory, hard disk,multimedia card micro type or card type memory (e.g., an SD or XDmemory, etc.), a RAM and a ROM. The image display device 100 may outputa file (a moving image file, a still image file, a music file, adocument, etc.) stored in the memory 160 back to provide the file to theuser and executes an application stored in the memory 160 to provide theapplication to the user.

The controller 180 controls the overall operation of the image displaydevice 100.

The controller 180 receives the screen information of the mobileterminal 200 from the mobile terminal 200 through the interface 150. Thecontroller 180 controls the signal processor 140 to process the screeninformation of the mobile terminal and output the processed video signalto the display 170.

The controller 180 receives the user command for controlling the mobileterminal 200 from the user through the user input unit 155. Thecontroller 180 may transmit data according to the received user commandto the mobile terminal 200 through the interface 150.

The user input unit 155 may include a keyboard, a mouse, etc. connectedto a wireless remote controller, a magic remote controller, and theimage display device 100. In the present specification, assume that theuser input unit 155 is a wireless remote controller.

FIG. 3 is a block diagram showing the internal configuration of themobile terminal 200 according to an embodiment of the present invention.

Referring to FIG. 3, the mobile terminal 200 includes a wirelesscommunication unit 210, an audio/video input unit 220, a sensing orsensor unit 240, an output unit 250, a memory 60, an interface 270, acontroller 280 and a power supply 190.

Components shown in FIG. 3 are not essential and the mobile terminal 200having more component or fewer components may be implemented.Hereinafter, the above-described components will be sequentiallydescribed.

The wireless communication unit 210 may include one or more modules forperforming wireless communication between the mobile terminal 200 and awireless communication system or between the mobile terminal 200 and anetwork in which the mobile terminal 200 is located. For example, thewireless communication unit 210 may include a broadcast reception module211, a mobile communication module 212, a wireless Internet module 213,a short-range communication module 214 and a location information module215.

The broadcast reception module 211 receives a broadcast signal and/orbroadcast related information from an external broadcast managementserver via a broadcast channel. The broadcast channel may include asatellite channel and a terrestrial channel. The broadcast managementserver may generate and transmit a broadcast signal and/or broadcastrelated information or receive a previously generated broadcast signaland/or broadcast related information to the terminal. The broadcastsignal may include a TV broadcast signal, a radio broadcast signal and adata broadcast signal and also include a combination of a TV broadcastsignal or radio broadcast signal and a data broadcast signal.

There is a variety of broadcast related information. For example, thebroadcast related information may take the form of an electronic programguide (EPG) of digital multimedia broadcasting (DMB) or an electronicservice guide (ESG) of digital video broadcast-handheld (DVB-H). Thebroadcast signal and/or broadcast related information received throughthe broadcast reception module 211 may be stored in the memory 260.

The mobile communication module 212 transmits and receives a radiosignal to and from at least one of a base station, an external terminaland a server over a mobile communication network. The radio signal mayinclude a variety of data according to voice signal, video call signalor text/multimedia message transmission and reception.

The wireless Internet module 213 accesses the wireless Internet and maybe mounted inside or outside the mobile terminal 200. For connection towireless Internet, Wireless Local Area Network (WLAN) (i.e., Wi-Fi),Wireless Broadband (WiBro), World Interoperability for Microwave Access(WiMax), and High Speed Downlink Packet Access (HSDPA) may be used.

The short-range communication module 214 performs short-rangecommunication. As short-range communication technologies, Bluetooth,radio frequency identification (RFID), infrared data association (IrDA),ultra wideband (UWB), ZigBee, NFC, etc. may be used.

The location information module 215 acquires the location of the mobileterminal and a representative example thereof includes a globalpositioning system (GPS) module.

Referring to FIG. 3, the A/V input unit 220 receives an audio signal ora video signal and may include a camera 221 and a microphone 222. Thecamera 221 processes an image frame such as a still image or a movingimage obtained by an image sensor in a video communication mode or acapture mode. The processed image frame may be displayed on the display251.

The image frame processed by the camera 221 may be stored in the memory260 or transmitted to an external device via the wireless communicationunit 210. Two or more cameras 221 may be included according to useenvironment. The microphone 222 receives and converts an external soundsignal in a communication mode, a recording mode or a voice recognitionmode into electrical voice data. In the communication mode, theprocessed voice data may be converted into a form transmittable to abase station to be output. Various noise elimination algorithms foreliminating noise generated in a process of receiving an external soundsignal may be implemented in the microphone 222.

The sensing unit 240 senses an opened/closed state of the mobileterminal 200, the position of the mobile terminal 200, contact ornon-contact of a user, the orientation of the mobile terminal andacceleration/deceleration of the mobile terminal and generates a sensorsignal for controlling operation of the mobile terminal 200. The sensingunit 240 may include a proximity sensor, a gravity sensor, a gyroscopesensor, etc.

The output unit 250 generates visual, audible or tactile output and mayinclude a display 251, an audio output module 252, an alarm module 253,a haptic module 254, etc.

The display 251 displays (outputs) information processed by the mobileterminal 200. For example, if the mobile terminal is in a communicationmode, a user interface (UI) or a graphical user interface (GUI)associated with communication is displayed. If the mobile terminal 200is in a video communication mode or a capture mode, captured and/orreceived image, UI or GUI is displayed. In addition, screen informationreceived from the external image display device 100 may be output.

The display 251 may include at least one of a liquid crystal display(LCD), a thin film transistor-liquid crystal display (TFT LCD), anorganic light emitting diode (OLED), a flexible display and a3-dimensional (3D) display.

The display 251 and a sensor (hereinafter, referred to as a “touchsensor”) for sensing a touch operation have a layered structure(hereinafter, referred to as a “touchscreen”), the display 251 mayfunctions as an input device and an output device. The touch sensor mayinclude a touch film, a touch sheet, a touch pad, etc.

The touch sensor may be configured to convert pressure applied to aspecific part of the display 251 or change in capacitance generated in aspecific part of the display 251 into an electrical input signal. Thetouch sensor may be configured to detect a touch location and area andtouch pressure.

If touch input to the touch sensor is present, a signal(s) correspondingthereto is sent to a touch controller. The touch controller processesthe signal(s) and then transmits a touch signal corresponding thereto tothe controller 280. The controller 280 determines which portion of thedisplay 251 is touched and the controller 280 analyzes a user commandaccording to the touch signal and controls the mobile terminal 200according to the user command.

The touch sensor may be configured to recognize a variety of touchoperations of a user. For example, the touch sensor may receive a usertouch varying according to touch pressure.

In implementation of the touch sensor, the touch sensor may include onetouch sensor for distinguishably recognizing weak touch pressure andstrong touch pressure or include a first touch sensor for recognizingweak touch pressure and a second touch sensor for recognizing strongtouch pressure.

The controller 280 may recognize the user command according to touchrecognized by the first touch sensor or the second touch sensor. Forexample, the controller 280 may recognize a user command correspondingto touch recognized by the first touch sensor as a first user commandand a user command corresponding to touch recognized by the second touchsensor as a second user command. The controller 280 controls the mobileterminal 200 according to the user command.

The audio output module 252 may output audio data received from thewireless communication unit 210 or stored in the memory 260 in a callsignal mode, a communication mode, a recording mode, a voice recognitionmode, a broadcast reception mode, etc. The audio output module 252 mayoutput a sound signal associated with a function (e.g., call signalreception sound, message reception sound, etc.) performed by the mobileterminal 200. Such an audio output module 252 may include a receiver, aspeaker, a buzzer, etc.

The alarm unit 253 outputs a signal notifying the user that an event hasoccurred in the mobile terminal 200. Examples of the event occurring inthe mobile terminal 200 include call signal reception, messagereception, key signal input, touch input, etc. The alarm unit 253 mayoutput a signal notifying the user that an event has occurred in a formother than a video signal or an audio signal, e.g., through vibrations.The video signal or the audio signal may be output through the display251 or the audio output module 252 and the display 251 and the audiooutput module 252 may serve as parts of the alarm unit 253.

The haptic module 254 generates a variety of tactile effects that theuser can feel. A typical example of the tactile effect generated by thehaptic module 254 is vibration. The strength and pattern of vibrationsgenerated by the haptic module 254 may be controlled. For example,different vibrations may be synthesized and output or may besequentially output.

The memory 260 may store a program for processing and control of thecontroller 280 and may temporarily store input or output data (forexample, a phonebook, messages, audio, still images, and moving images).The memory 260 may store data about vibrations and sound of variouspatterns output upon touch input of the touchscreen.

The interface 270 includes an external communication unit 271 and a userinput unit 272 and serves as an interface with all external devicesconnected to the mobile terminal 200.

The external device communication unit 271 may receive data from allexternal devices connected to the mobile terminal 200 or transmit dataof the mobile terminal 200 to an external device.

The user input unit 272 may generate input data for enabling the user tocontrol operation of the mobile terminal 200. The user input unit 272may include a key pad, a dome switch, a jog wheel, a jog switch, etc.The user input unit 272 may include the above-described touch pad(static pressure/static electrical).

The controller 280 controls the overall operation of the mobile terminal200. For example, the controller 280 performs control and processingassociated with voice communication, data communication, videocommunication, and the like.

The power supply 290 receives external power or internal power andsupplies power required for operation to each component under control ofthe controller 280.

The various embodiments described herein may be implemented in arecording medium readable by a computer or a computer-like device byvarious means, for example, hardware, firmware, software, or acombination thereof. In a hardware configuration, the embodiments of thepresent invention may be achieved by one or more Application SpecificIntegrated Circuits (ASICs), Digital Signal Processors (DSPs), DigitalSignal Processing Devices (DSPDs), Programmable Logic Devices (PLDs),Field Programmable Gate Arrays (FPGAs), processors, controllers,microcontrollers, microprocessors, and other electrical units forperforming functions. In some cases, the embodiments of the presentinvention may be implemented in the controller 180. In a softwareconfiguration, procedures and functions according to the embodiments ofthe present invention may be implemented in the form of software modulesthat perform one or more functions and operations described herein.Software code may be realized by a software application written in anappropriate programming language, stored in the memory 260, and executedby the controller 180.

Hereinafter, a user interface provision method using the image displaydevice 100 and the mobile terminal 200 will be described.

First, FIGS. 4 to 7 are diagrams illustrating a user interface provisionmethod according to a first embodiment of the present invention. Inparticular, FIGS. 4 to 6 show a multi mirroring service for outputtingthree screens 101, 102 and 103 of three mobile terminals 201, 202 and203 via the image display device 100 if the three mobile terminals 201,202 and 203 are connected to the image display device 100.

That is, the three mobile terminals 201, 202 and 203 transmit respectivescreen information thereof to the image display device 100 and the imagedisplay device 100 processes and outputs the screen information of themobile terminals 201, 202 and 203 on the display 170.

As shown in the figures, the image display device 100 outputs the screenof the first mobile terminal 201 as a first screen 101, the screen ofthe second mobile terminal 202 as a second screen 102 and the screen ofthe third mobile terminal 203 as a third screen 103.

FIG. 4 also shows the three mobile terminals 201, 202 and 203 areconnected to the image display device 100 through wired technologiessuch as Ethernet, universal serial bus (USB) or mobile high-definition(MHL). FIG. 5 shows the three mobile terminals 201, 202 and 203 areconnected to the image display device 100 through wireless technologiessuch as Wi-Fi or WiDi. FIG. 6 shows the three mobile terminals 201, 202and 203 are connected to the image display device 100 through wired orwireless technologies such as USB, MHL or WiDi.

Accordingly, the mobile terminals 200 may be connected to the imagedisplay device 100 using various connection methods to transmit screeninformation thereof to the image display device 100. The image displaydevice 100 outputs the received screen information on the display 170.

FIG. 7 shows a multi mirroring service for outputting three screens 101,102 and 103 of three mobile terminals 201, 202 and 203 and an image 104of the image display device 100 via the image display device 100 if thethree mobile terminals 201, 202 and 203 are connected to the imagedisplay device 100.

As shown in FIG. 7, the image display device 100 outputs the screen ofthe first mobile terminal 201 as a first screen 101, the screen of thesecond mobile terminal 202 as a second screen 102, the screen of thethird mobile terminal 203 as a third screen 103, and the image of theimage display device 100 as a fourth screen 104.

As described above, a user command input to the image display device 100may be transmitted to the mobile terminal 200 through data communicationbetween the image display device 100 and the mobile terminal 200. Thatis, if the user enters a user command to the image display device 100for outputting the screen of the mobile terminal 200, the image displaydevice 100 transmits the received user command to the mobile terminal200 and the mobile terminal 200 performs operation of the mobileterminal 200 according to the received user command.

As shown in the figure, if several mobile terminals 200 are connected tothe image display device 100, the user can control the mobile terminal200 via the screen of the mobile terminal 200 output on the imagedisplay device 100 using the user input unit 155.

That is, if the user enters a user command to the screen of the mobileterminal 200 to be controlled using the remote controller 155, the imagedisplay device 100 transmits data corresponding to the received usercommand to the mobile terminal 200.

Accordingly, the user can control the first mobile terminal 201 via thefirst screen 101, control the second mobile terminal 202 via the secondscreen 102, control the third mobile terminal 203 via the third screen103, and control the image display device 100 via the fourth screen 104.

Accordingly, the image display device 100 determines on which screen theuser enters a user command or which mobile terminal is controlled by theuser. That is, the image display device 100 determines a target mobileterminal 200 which is desired to be controlled by the user. The imagedisplay device 100 transmits the received user command to the targetmobile terminal 200 to be controlled by the user.

A method of determining the target mobile terminal 200 will be describedin detail later with reference to FIGS. 11 to 13.

Next, FIGS. 8 to 10 are diagrams illustrating a screen configurationmethod of the image display device 100 for providing a multi-mirroringservice according to an embodiment of the present invention.

In the above-described multi mirroring service, the screens of severalmobile terminals 200 are output via the image display device 100.Accordingly, the image display device 100 arranges the screens of themobile terminals 200 such that the user intuitively views the screens ofthe mobile terminals 200. Hereinafter, the method of arranging thescreens of the mobile terminals 200 will be described.

First, the image display device 100 can arrange the screens of themobile terminals 200 according to a predetermined scenario. That is, iftwo mobile terminals 200 are connected to the image display device 100,the screen of the image display device 100 may be split into right andleft screens, the screen 101 of the first mobile terminal 201 may beoutput on the left screen, and the screen 102 of the second mobileterminal 202 may be output on the right screen.

If a new third mobile terminal 203 is further connected to the imagedisplay device 100, the image display device 100 may output the screen103 of the third mobile terminal 203 in a picture-in-picture (PiP)format.

If a new fourth mobile terminal is further connected to the imagedisplay device 100, the screen of the image display device 100 isquartered to output the screens of the mobile terminals 200.

The image display device 100 may arrange the screens of the mobileterminals 200 connected to the image display device 100 according to apredetermined scenario.

The image display device 100 can also receive a user command and changethe arrangement of the screens of the mobile terminals 200. That is, thepresent invention provides a user interface for changing the locationand size of the screens of the mobile terminals 200.

Accordingly, the image display device 100 receives a user commandcorresponding to change in screens of the mobile terminals 200 andchanges the screens of the mobile terminals 200 according to thereceived user command.

In particular, FIGS. 8 to 10 show examples of changing the arrangementof the screens of the mobile terminals 200 by entering a user command tothe image display device 100.

FIG. 8 shows the user changing the locations of the screens of themobile terminals 200. The screen 101 of the first mobile terminal 201 isoutput on a main screen and the other screens 102, 103 and 104 areoutput on a sub screen.

At this time, when the user enters a user command for dragging anddropping the second screen 102 to the first screen 101, the imagedisplay device 100 outputs the second screen 102 on the main screen andoutputs the first screen 101 at the location of the second screen 102.That is, the image display device 100 changes the locations of the firstscreen 101 and the second screen 102.

The user command may be input using various methods such as selection ofscreens, the locations of which will be changed, in addition to adrag-and-drop method.

FIG. 9 shows the user changing the sizes of the screens of the mobileterminals 200. The first to fourth screens 101 to 104 are output on theimage display device 100.

At this time, when the user enters a user command for changing the sizeof the first screen, the image display device 100 changes the size ofthe first screen according to the user command.

The image display device 100 changes the screens of the mobile terminals200 according to the user command. However, the image display device 100of the present invention can also change the arrangement of the screensof the mobile terminals 200 when an event has occurred in the mobileterminal 200.

In particular, FIG. 10 shows the image display device 100 changingarrangement of the screens of the mobile terminals 200 in order tonotify the user that an event has occurred in the mobile terminal 200.

In FIG. 10, an event has occurred in the third mobile terminal 203. Theimage display device 100 thus arranges the third screen 103corresponding to the third mobile terminal 203 at the foremost side ofthe screen and highlights the third screen 103 in order to notify theuser that the event has occurred.

Thus, the user may perceive that the event has occurred in the thirdmobile terminal 203 and easily control the third mobile terminal 203 viathe third screen 103.

The image display device 100 determines the screen to which the usercommand is input, that is, the screen of the target mobile terminal 200to be controlled by the user. This is because the image display device100 should send the received user command to the target mobile terminal200 to be controlled by the user.

Next, FIGS. 11 to 13 are diagrams illustrating a first embodiment of amulti UIBC service providing method of the present invention.

In the first embodiment of the multi UIBC service providing method ofthe present invention, a method of efficiently determining the targetmobile terminal 200 to be controlled by the user using locationinformation of the screens and location information of a cursor at theimage display device 100 is provided.

FIG. 11 shows two mobile terminals 200 are connected to the imagedisplay device 100, the first mobile terminal 201 is output on the firstscreen 101 and the second mobile terminal 202 is output on the secondscreen 102.

The image display device 100 generates boundary coordinate informationof the first screen 101 and boundary coordinate information of thesecond screen 102. The boundary coordinate information is location andsize information of a screen. The boundary coordinate information may becomposed of coordinates of boundary points of the screen or coordinatesof an original point of the screen and width/length information of thescreen.

If the user points to the image display device 100 using the wirelessremote controller 155 in order to control the mobile terminal 200, theimage display device 100 outputs a cursor which is a guide image 156 ata point pointed to by the wireless remote controller 155. The imagedisplay device 100 generates coordinate information of the cursor 156output at the point pointed to by the wireless remote controller 155.

The image display device 100 compares the boundary coordinateinformation of the first screen 101 and the second screen 102 with thecoordinate information of the cursor 156 and determines which screen ispointed to by the user.

In FIG. 11, the user moves the cursor 156 from the second screen 102 tothe first screen 101. If the cursor 156 is located on the second screen,the image display device 100 may determine that the mobile terminal 200to be controlled by the user is the second mobile terminal 202 and, ifthe cursor 156 is located on the first screen, the image display device100 may determine that the mobile terminal 200 to be controlled by theuser is the first mobile terminal 201.

That is, the image display device 100 compares the boundary coordinateinformation of the mobile terminal 200 with the coordinate informationof the cursor 156 of the wireless remote controller 155, determines thescreen on which the cursor 156 is located, and determines the mobileterminal 200 corresponding to the determined screen as the target mobileterminal 200 to be controlled by the user.

Accordingly, the image display device 100 controls the interface 150 toestablish communication in order to transmit and receive data to andfrom the mobile terminal 200 which is determined as the target mobileterminal 200.

FIG. 12 shows three mobile terminals 200 are connected to the imagedisplay device 100, the first mobile terminal 201 is output on the firstscreen 101, the second mobile terminal 202 is output on the secondscreen 102, and the third mobile terminal 203 is output on the thirdscreen 103.

The image display device 100 generates boundary coordinate informationof the first screen 101, boundary coordinate information of the secondscreen 102 and boundary coordinate information of the third screen 103.The boundary coordinate information is location and size information ofa screen. The boundary coordinate information may be composed ofcoordinates of boundary points of the screen or coordinates of anoriginal point of the screen and width/length information of the screen.

If the user points to the image display device 100 using the wirelessremote controller 155 in order to control the mobile terminal 200, theimage display device 100 outputs a cursor which is a guide image 156 ata point pointed to by the wireless remote controller 155. The imagedisplay device 100 generates coordinate information of the cursor 156output at the point pointed to by the wireless remote controller 155.

The image display device 100 compares the boundary coordinateinformation of the first screen 101, the second screen 102 and the thirdscreen 103 with the coordinate information of the cursor 156 anddetermines which screen is pointed to by the user.

FIG. 12 shows the user moves the cursor 156 from the third screen 103 tothe second screen 102. If the cursor 156 is located on the third screen103, the image display device 100 may determine that the mobile terminal200 to be controlled by the user is the third mobile terminal 203 and,if the cursor 156 is located on the second screen, the image displaydevice 100 may determine that the mobile terminal 200 to be controlledby the user is the second mobile terminal 202.

That is, the image display device 100 compares the boundary coordinateinformation of the mobile terminal 200 with the coordinate informationof the cursor 156 of the wireless remote controller 155, determines thescreen on which the cursor 156 is located, and determines the mobileterminal 200 corresponding to the determined screen as the target mobileterminal 200 to be controlled by the user.

FIG. 13 shows four mobile terminals 200 are connected to the imagedisplay device 100, the first mobile terminal 201 is output on the firstscreen 101, the second mobile terminal 202 is output on the secondscreen 102, the third mobile terminal 203 is output on the third screen103, and the fourth mobile terminal 204 is output on the fourth screen104. The image display device 100 generates boundary coordinateinformation of the first screen 101 to the fourth screen 104.

FIG. 13 shows the user moves the cursor 156 from the fourth screen 104to the first screen 101. As described above, if the cursor 156 islocated on the fourth screen, the image display device 100 may determinethe mobile terminal 200 to be controlled by the user as the fourthmobile terminal 204 and, if the cursor 156 is located on the firstscreen, the image display device 100 may determine the mobile terminal200 to be controlled by the user as the first mobile terminal 201.

The image display device 100 compares the boundary coordinateinformation of the mobile terminal 200 with the coordinate informationof the cursor 156 of the wireless remote controller 155, determines thescreen on which the cursor 156 is located, and determines the mobileterminal 200 corresponding to the determined screen as the target mobileterminal 200 to be controlled by the user.

FIGS. 14 to 15 are diagrams illustrating a second embodiment of a multiUIBC service providing method of the present invention.

In the second embodiment of the multi UIBC service providing method ofthe present invention, the image display device 100 rapidly establishescommunication with the changed target mobile terminal 200.

In particular, FIG. 14 shows two mobile terminals 200 are connected tothe image display device 100, the first mobile terminal 201 is output onthe first screen 101 and the second mobile terminal 202 is output on thesecond screen 102.

As described above, the image display device 100 generates boundarycoordinate information of the first screen 101 and boundary coordinateinformation of the second screen 102. The boundary coordinateinformation is location and size information of a screen. The boundarycoordinate information may be composed of coordinates of boundary pointsof the screen or coordinates of an original point of the screen andwidth/length information of the screen.

The image display device 100 may generate information about adjacentscreens of the first screen 101 and the second screen 102. For example,adjacent screen connection regions 301 and 302 may be set oridentification of the adjacent screens may be generated. The adjacentscreen connection regions 301 and 302 are regions extending from theboundary with adjacent screens. The image display device 100 generatescoordinate information of the adjacent screen connection regions andidentification of the adjacent screens.

Unlike FIG. 14, as shown in FIG. 15, the present invention is applicableto the screen being split into three screen regions. As shown in FIG.15, a third screen 103 is only adjacent to the second screen 102, and isnot adjacent to the first screen 101. Accordingly, identification of anadjacent screen does not need to be generated between the third screen103 and the first screen 101.

The first adjacent screen connection region 301 is set on the firstscreen 101 and corresponds to a predetermined region from the boundarywith the second screen 102 adjacent to the first screen 101. The firstadjacent screen connection region 301 includes information about thesecond screen 102 which is an adjacent screen of the first screen 101.

The second adjacent screen connection region 302 is set on the secondscreen 102 and corresponds to a predetermined region from the boundarywith the first screen 101 adjacent to the second screen 102. The secondadjacent screen connection region 302 includes information about thefirst screen 101 adjacent to the second screen 102.

As shown in FIG. 14, if the user moves the wireless remote controller155 from the second screen 102 to the first screen 101 in order tochange the target mobile terminal 200, the cursor moves from the secondscreen 102 to the first screen 101 first passing through the secondadjacent screen connection region 302.

At this time, the image display device 100 determines whether the cursoris located in the adjacent screen connection regions 301 and 302. If thecursor is located in the adjacent screen connection regions 301 and 302,the image display device 100 establishes communication with the mobileterminal 200 corresponding to the adjacent screen connection regions 301and 302.

In more detail, as shown in FIG. 14, if the user moves the wirelessremote controller 155 from the second screen 102 to the first screen 101in order to change the target mobile terminal 200 to the first mobileterminal 201, the cursor 156 first passes through the second adjacentscreen connection region 302.

Thus, when the cursor 156 is located in the second adjacent screenconnection region 302, the image display device 100 prepares to providea UIBC service for the first screen 101 adjacent to the second screen102. That is, the image display device 100 establishes datacommunication with the first mobile terminal 201 corresponding to thefirst screen 101 when the cursor 156 first moves into the secondconnection region 302 included in the second screen 102.

The second screen 102 in FIG. 14 corresponds to the second mobileterminal 102. However, the second screen 102 could be a screen displayedby the display device 100 itself (e.g., a broadcast program).

If the user changes the target mobile terminal 200, it is possible torapidly establish communication with the changed target mobile terminal200 and to efficiently provide the UIBC service to the user.

In FIG. 14, the cursor 156 may pass through the second adjacent screenconnection region 302 and be located in the first adjacent screenconnection region 301. In this instance, the image display device 100may ignore the cursor 156 being previously located in the first adjacentscreen connection region 301.

FIG. 15 shows three mobile terminals 200 are connected to the imagedisplay device 100, the first mobile terminal 201 is output on the firstscreen 101, the second mobile terminal 202 is output on the secondscreen 102 and the third mobile terminal 203 is output on the thirdscreen 103.

As described above, the image display device 100 generates boundarycoordinate information of the first screen 101, the second screen 102and the third screen 103 and sets the adjacent screen connection regions301, 302 and 303 of the first screen 101, the second screen 102 and thethird screen 103.

The adjacent screen connection regions 301, 302 and 303 arepredetermined regions extending from the boundaries of adjacent screensand include coordinate information of the regions and information aboutadjacent screens.

The first adjacent screen connection region 301 is set on the firstscreen 101 and corresponds to a predetermined region extending from theboundary with the second screen 102 adjacent to the first screen 101.The first adjacent screen connection region 301 includes informationabout the second screen 102 which is the adjacent screen of the firstscreen.

The second adjacent screen connection region 302 is set on the secondscreen 102 and corresponds to a predetermined region extending from theboundary with the first screen 101 adjacent to the second screen 102.The second adjacent screen connection region 302 includes informationabout the first screen 101 which is the adjacent screen of the secondscreen.

The third adjacent screen connection region 303 is set on the thirdscreen in the third screen 103 and corresponds to a predetermined regionextending from the boundary with the second screen 102 adjacent to thethird screen 103. The third adjacent screen connection region 303includes information about the second screen 102 which is the adjacentscreen of the third screen 103.

As shown in FIG. 15, if the user moves the wireless remote controller155 from the third screen 103 to the second screen 102 in order tochange the target mobile terminal 200, the cursor moves from the thirdscreen 103 to the second screen 102 through the third adjacentconnection region 303.

At this time, the image display device 100 determines whether the cursoris located in the adjacent screen connection regions 301, 302 and 303and establishes communication with the mobile terminals 200corresponding to the adjacent screen connection regions 301, 302 and 303if the cursor is located in the adjacent screen connection regions 301,302 and 303.

In FIG. 15, if the user moves the wireless remote controller 155 fromthe third screen 103 to the second screen 102 in order to change thetarget mobile terminal 200 to the second mobile terminal 202, the cursor156 passes through the third adjacent screen connection region 303.

If the cursor 156 is located in the third adjacent screen connectionregion 303, the image display device 100 prepares to provide a UIBCservice for the second screen 102 adjacent to the third screen 103. Thatis, the image display device 100 establishes data communication with thesecond mobile terminal 202 corresponding to the second screen 102.

Next, FIG. 16 is a flowchart illustrating a user interface provisionmethod according to a first embodiment of the present invention.

The user interface provision method according to the first embodiment ofthe present invention includes determining whether the image displaydevice 100 is providing a multi mirroring service (S100), generating andstoring boundary coordinate information of multiple screens (S200),setting the respective adjacent screen connection regions 300 of themultiple screens (S300), reading the coordinates of the cursor 156 ofthe image display device 100 (S400), determining whether the coordinatesof the cursor 156 correspond to the adjacent screen connection regions300 (S500), establishing data communication with the mobile terminal 200corresponding to an adjacent screen (S600), comparing the coordinates ofthe cursor 156 with boundary coordinate information to determine onwhich screen a UIBC operation is performed (S700), and performing theUIBC operation with the mobile terminal 200 corresponding to thedetermined screen (S800).

First, the image display device 100 determines whether the multimirroring service is being provided (S100).

As described above, the multi mirroring service indicates that severalmobile terminals 200 are connected to one image display device 100 tosimultaneously output the screens of the mobile terminals 200 via theimage display device 100. The user interface method of the presentinvention provides a multi UIBC service configured to allow the user tocontrol several mobile terminals 201 and 202 via the image displaydevice 100 if such a multi mirroring service is provided.

The image display device 100 stores boundary coordinate information ofthe multiple screens (S200). The boundary coordinate information may becomposed of coordinates of boundary points of the screen or coordinatesof an original point of the screen and width/length information of thescreen.

The image display device 100 sets the adjacent screen connection regions300 of the multiple screens (S300). The adjacent screen connectionregions 301 and 302 are regions extending from the boundary withadjacent screens and include coordinate information of the correspondingregion and information about the adjacent screen.

The image display device 100 reads the coordinates of the cursor 156 ofthe image display device 100 (S400).

If the user points to the image display device 100 using the wirelessremote controller 155 in order to control the mobile terminal 200, theimage display device 100 outputs the cursor which is the guideinformation 156 at the point pointed to by the wireless remotecontroller 155. The image display device 100 generates and reads thecoordinate information of the cursor output at the point pointed to bythe wireless remote controller 155.

The image display device 100 determines whether the coordinates of thecursor 156 correspond to the adjacent screen connection regions 300(S500). If the cursor 156 is located in the adjacent screen connectionregions 301 and 302, the image display device 100 establishescommunication with the mobile terminal 200 corresponding to the adjacentscreen in order to perform UIBC of the adjacent screen (S600).

The image display device 100 compares the coordinates of the cursor 156with the boundary coordinate information to determine the screen onwhich the UIBC operation is performed (S700).

That is, the image display device 100 compares the boundary coordinateinformation of each screen with the coordinate information of the cursor156, determines the screen on which the cursor is located, anddetermines the mobile terminal 200 corresponding to the determinedscreen as the target mobile terminal 200 to be controlled by the user.

Finally, the image display device 100 performs the UIBC operation withthe mobile terminal 200 corresponding to the determined screen (S800).

Thereafter, a user interface provision method for allowing a user toefficiently control the shared screens if the screens of the imagedisplay device 100 and the mobile terminals 200 are shared will bedescribed.

FIGS. 17 and 18 are flowcharts illustrating a user interface provisionmethod according to a second embodiment of the present invention.

First, referring to FIG. 17, the mobile terminal 200 determines whethera user touch is input (S101). The controller 280 of the mobile terminal200 determines whether the user touch is input to the touch sensor.

If the user touch is not input to the touch sensor, the controller 280captures a current screen and generates screen information of the mobileterminal 200 (S106). The controller 280 transmits the generated screeninformation to the image display device 100 via the external devicecommunication unit 271 (S107).

If the user touch is input to the touch sensor, the controller 280determines whether the first user touch is input to the touch sensor(S102). That is, the controller 280 determines whether the user touchinput to the touch sensor is a first user touch.

As described above, the touch sensor may distinguishably sense thereceived user touch as a first user touch or a second user touch. Thetouch sensor includes a first touch sensor for sensing a first usertouch and a second touch sensor for sensing a second user touch.

In the present invention, a user touch may be divided into a first usertouch and a second user touch according to various criteria. That is,the touch sensor may distinguishably sense a first user touch and asecond user touch according to various criteria.

For example, the touch sensor may recognize a touch of a small region asthe first user touch and recognize a touch of a large region as a seconduser touch. The touch sensor may recognize a touch of a short period oftime as the first user touch and recognize a touch of a long period oftime as the second user touch. The touch sensor may recognize a touch ofweak pressure as the first user touch and recognize a touch of strongpressure as the second user touch. The touch sensor may recognize atouch of a specific gesture as the first user touch and a touchthereafter as the second user touch.

At this time, the controller 280 recognizes the first user touch as afirst user command and recognizes the second user touch as a second usercommand and controls the mobile terminal 200 according to the usercommand.

In the present specification, assume that the first user touch is atouch of weak pressure and the second user touch is a touch of strongpressure.

Accordingly, the controller 280 determines whether a first user touch isinput using the first touch sensor. That is, the controller 280determines whether a first user touch which is a touch of weak pressureis input using the first touch sensor.

If the first user touch is input to the touch sensor, the controller 280controls the output unit 250 to output a guide image 156 in a screenregion in which the first user touch is input (S103).

The guide image 156 indicates the screen region in which the first usertouch is sensed and may be output in the form of a pointer, an icon,etc. In the present specification, for example, the controller 280outputs a pointer image as the guide image 156.

Accordingly, if the user inputs a first user touch in a specific regionof the screen, the controller 280 outputs the guide image 156 in thescreen region in which the first user touch is input.

If the first user touch is not input to the touch sensor, the controller280 determines whether a second user touch is input to the touch sensor(S104). That is, the controller 280 determines whether a second usertouch which is a user touch of strong pressure is input using the secondtouch sensor.

If the second user touch is input to the touch sensor, the controller280 performs a user command corresponding to the second user touch(S105). That is, the controller 280 analyzes the user commandcorresponding to second user touch and controls the mobile terminals 200according to the user command.

Finally, the controller 280 captures a current screen and generatesscreen information of the mobile terminal 200 (S106). The controller 280transmits the generated screen information to the image display device100 via the external device communication unit 271 (S107).

FIG. 18 is a flowchart illustrating a user interface provision methodusing the mobile terminals 200 and the image display device 100according to another embodiment of the present invention.

First, the controller 280 determines whether the mobile terminals 200are connected to the image display device 100 for data communication soas to provide a mirroring service for transmitting the screens of themobile terminals 200 to the image display device 100 (S201).

If the mirroring service is being provided, the controller 280 sets avoice command set corresponding to functions which are being executed bythe mobile terminals 200 (S202). For example, if moving-image navigationis executed on the mobile terminal 200, the controller 280 maypreviously set a voice command set of “rewind”, “fast forward”, “play”and “stop” corresponding to moving-image navigation functions. Inaddition, if an Internet browser is being executed on the mobileterminal 200, the controller 280 may previously set a voice command setof “selection”, “click” and “back” corresponding to the functions of theInternet browser, such that the controller 280 recognizes onlypredetermined voice commands so as to rapidly perform the functions.

The controller 280 determines whether a predetermined voice command isreceived (S203). The controller 280 may enter a voice recognition modein order to recognize the predetermined voice command. The controller280 may output a screen corresponding to a voice standby mode via theoutput unit 250, in order to notify the user of entrance into the voicestandby mode.

If a voice command is received, the controller 280 performs a functioncorresponding to the received voice command (S204). The controller 280analyzes a user command corresponding to the received voice command andexecutes the user command.

If the voice command is not received, the controller 280 performs theuser interface provision method described with reference to FIG. 4 (S101to S107). That is, if the user inputs a first user touch in a specificregion of the screen, the controller 280 outputs the guide image 156 inthe screen region in which the first user touch is input. If the userinputs a second user touch, the controller 280 executes the user commandcorresponding to the second user touch.

The user interface provision method may be implemented using variousembodiments. Hereinafter, user interfaces according to variousembodiments of the present invention will be described.

Next, FIGS. 19 to 23 are diagrams illustrating a first embodiment of auser interface provision method for controlling the screen of an imagedisplay device 100 of the present invention.

FIG. 19 shows a user interface environment according to the firstembodiment of the user interface provision method for controlling thescreen of the image display device 100 of the present invention.

The mobile terminal 200 transmits the screen information of the mobileterminal 200 to the image display device 100 through data communication.The image display device 100 outputs the received screen information ofthe mobile terminal 200 to the user.

That is, the image display device 100 receives and outputs the screeninformation of the mobile terminal 200, such that the image displaydevice 100 and the mobile terminal 200 output the same screen. Thus, theuser may receive the screen of the mobile terminal 200 from the mobileterminal 200 and the image display device 100.

In FIG. 19, an application icon screen which is the screen of the mobileterminal 200 is transmitted to the image display device 100 and isequally output on the image display device 100.

FIG. 20 shows the user inputting a first user touch to the mobileterminal 200 and FIG. 21 shows the screen of the image display device100 if a first user touch is input to the mobile terminal 200.

In FIG. 20, the user inputs a first user touch in one right lower regionof the mobile terminal 200. Then, the user drags the first user touch toa region in which an application icon “Web” is displayed. That is, theuser inputs a touch of weak pressure in one right lower region of themobile terminal 200 and then drags the first user touch to a region inwhich an application icon “Web” is displayed.

At this time, the controller 280 of the mobile terminal 200 controls theoutput unit 250 and outputs the guide image 156 in the screen region inwhich the first user touch is input.

As described above, the guide image 156 indicates the screen region inwhich the first user touch is sensed. In FIG. 21, the controller 280outputs a pointer as the guide image 156. Accordingly, if the userinputs the first user touch in one region of the screen, the controller280 outputs the pointer in the screen region in which the first usertouch is input.

As shown in FIG. 20, if the user inputs the first user touch in oneright lower region of the mobile terminal 200 and then drags the firstuser touch to a region in which an application icon “Web” is displayed,the controller 280 outputs the guide image 156 according to the regionin which the first user touch is input.

The controller 280 captures a current screen of the mobile terminal 200,generates screen information of the mobile terminal 200, and transmitsthe generated screen information to the image display device 100 via theexternal device communication unit 271. The transmitted screeninformation of the mobile terminal 200 is information about the screenon which the guide image 156 according to the first user touch isfurther displayed.

The image display device 100 outputs the screen information of themobile terminal 200 on the display 170.

FIG. 21 shows the output of the screen information of the mobileterminal 200 transmitted to the image display device 100, in which theguide image 156 according to the first user touch is further displayed.That is, the pointer which is the guide image 156 is output while movingfrom the right lower region, in which the first user touch is input, ofthe screen to the region in which the application icon “Web” isdisplayed.

FIG. 22 shows the user inputting a second user touch to the mobileterminal 200 and FIG. 23 shows the screen of the image display device100 if a second user touch is input to the mobile terminal 200.

In FIG. 22, the user inputs the second user touch in the region, inwhich an application icon “Web” is displayed, of the mobile terminal200. That is, the user inputs a touch of a strong pressure in the regionin which an application icon “Web” is displayed.

At this time, the controller 280 of the mobile terminal 200 executes theuser command corresponding to the second user touch. That is, thecontroller 280 analyzes the user command corresponding to the seconduser touch and controls the mobile terminal 200 according to the usercommand.

Accordingly, the controller 280 analyzes the user command correspondingto the second user touch as execution of the application “Web” and thusexecutes the application “Web”.

The controller 280 captures a current screen of the mobile terminal 200,generates screen information of the mobile terminal 200, and transmitsthe generated screen information to the image display device 100 via theexternal device communication unit 271. The transmitted screeninformation of the mobile terminal 200 is information about the screenon which the application “Web” is executed.

The image display device 100 outputs the screen information of themobile terminal 200 on the display 170.

FIG. 23 shows the output of the screen information of the mobileterminal transmitted to the image display device 100, in which theapplication “Web” is executed. That is, the screen of the application“Web” executed according to the second user touch is output.

In control of the screen output on the image display device 100 usingthe mobile terminal 200, the user may manipulate the mobile terminal 200in a state of fixing their eyes on the image display device 100, therebyincreasing user convenience. That is, it is possible to solve userinconvenience in which the user should manipulate the mobile terminal200 while alternately viewing the mobile terminal 200 and the imagedisplay device 100.

In display of the guide image 156 corresponding to the user command onthe screen, a complicated protocol for calculating the input location ofthe user command does not need to be mounted between devices and thuscomplexity of the user interface can be significantly reduced.

Next, FIGS. 24 to 28 are diagrams illustrating a second embodiment of auser interface provision method for controlling the screen of an imagedisplay device 100 of the present invention.

FIG. 24 shows a user interface environment according to the secondembodiment of the user interface provision method for controlling thescreen of the image display device 100 of the present invention.

The mobile terminal 200 transmits the screen information of the mobileterminal 200 to the image display device 100 through data communication.The image display device 100 outputs the screen information of themobile terminal 200 to the user.

That is, the image display device 100 receives and outputs the screeninformation of the mobile terminal 200, such that the image displaydevice 100 and the mobile terminal 200 output the same screen. Thus, theuser may receive the screen of the mobile terminal 200 from the mobileterminal 200 and the image display device 100.

In FIG. 24, a moving image screen which is the screen of the mobileterminal 200 is transmitted to the image display device 100 and isequally output on the image display device 100.

FIG. 25 shows the user inputting a first user touch to the mobileterminal 200 and FIG. 26 shows the screen of the image display device100 if a first user touch is input to the mobile terminal 200.

In FIG. 25, the user inputs a first user touch in one right lower regionof the mobile terminal 200. Then, the user drags the first user touch toa “rewind” menu region. That is, the user inputs a touch of weakpressure in one right lower region of the mobile terminal 200 and thendrags the first user touch to a “rewind” menu region.

If the first user touch is input to the mobile terminal 200, thecontroller 280 of the mobile terminal 200 executes the user commandcorresponding to the first user touch. That is, the controller 280 mayanalyze the user command corresponding to the second user touch andcontrols the mobile terminal 200 according to the user command.

If a first user touch is input to the mobile terminal 200 when a movingimage is played back on the mobile terminal 200 as shown in FIG. 25, thecontroller 280 may recognize the first user touch as a user command forexecuting moving-image navigation. The controller 280 executes the usercommand corresponding to the first user touch and controls the outputunit 250 to output a moving-image navigation screen.

The controller 280 controls the output unit 250 to output the guideimage 156 in the screen region in which the first user touch is input.As shown in FIG. 25, if the user inputs the first user touch in oneregion of the screen, the controller 280 outputs the pointer 300 in thescreen region in which the first user touch is input.

As shown in FIG. 25, if the user inputs the first user touch in oneright lower region of the mobile terminal 200 and then drags the firstuser touch to the “rewind” menu region, the controller 280 outputs theguide image 156 along the region in which the first user touch is input.

The controller 280 captures a current screen of the mobile terminal 200,generates screen information of the mobile terminal 200, and transmitsthe generated screen information to the image display device 100 via theexternal device communication unit 271.

For example, if the mobile terminal 200 and the image display device 100are connected via Wi-Fi, the controller 280 transmits the screeninformation of the mobile terminal 200 to the image display device 100via Wi-Fi data communication. Then, the controller 280 encodes thescreen information of the mobile terminal 200 and transmits the encodedscreen information to the image display device 100. The image displaydevice 100 decodes the received screen information and outputs thedecoded screen information.

The screen information of the mobile terminal 200 is information aboutthe screen on which the user command corresponding to the first usertouch is executed and the guide image 156 according to the first usertouch is further displayed.

The image display device 100 outputs the screen information of themobile terminal 200 on the display 170.

FIG. 26 shows the output of the screen information of the mobileterminal 200 transmitted to the image display device 100, in which amoving-image navigation screen is output and the guide image 156according to the first user touch is further displayed. The pointerwhich is the guide image 156 is output while moving from the right lowerregion of the screen, in which the first user touch is input, to the“rewind” menu region.

FIG. 27 shows the user inputting a second user touch to the mobileterminal 200 and FIG. 28 shows the screen of the image display device100 if the second user touch is input to the mobile terminal 200.

In FIG. 27, the user inputs the second user touch in the “rewind” menuregion of the mobile terminal 200. That is, the user inputs a touch ofstrong pressure in the “rewind” menu region.

At this time, the controller 280 of the mobile terminal 200 executes theuser command corresponding to the second user touch. That is, thecontroller 280 may analyze the user command corresponding to the seconduser touch and controls the mobile terminal 200 according to the usercommand.

Accordingly, the controller 280 executes the “rewind” menu according tothe user command corresponding to the second user touch. The controller280 captures a current screen of the mobile terminal 200, generatesscreen information of the mobile terminal 200, and transmits thegenerated screen information to the image display device 100 via theexternal device communication unit 271. The transmitted screeninformation is information about the screen on which the “rewind” menuof the moving image is executed.

The image display device 100 outputs the screen information of themobile terminal 200 on the display 170.

FIG. 28 shows the output of the screen information of the mobileterminal 200 transmitted to the image display device 100, in which the“rewind” menu is executed. That is, the screen on which the “rewind”menu is executed according to the second user touch is output.

Next, FIGS. 29 to 33 are diagrams illustrating a third embodiment of auser interface provision method for controlling the screen of an imagedisplay device of the present invention.

FIG. 29 shows a user interface environment according to the thirdembodiment of the user interface provision method for controlling thescreen of the image display device 100 of the present invention.

The mobile terminal 200 transmits the screen information of the mobileterminal 200 to the image display device 100 through data communication.The image display device 100 outputs the screen information of themobile terminal 200 to the user.

That is, the image display device 100 receives and outputs the screeninformation of the mobile terminal 100, such that the image displaydevice 100 and the mobile terminal 200 output the same screen. Thus, theuser may receive the screen of the mobile terminal 200 from the mobileterminal 200 and the image display device 100.

In FIG. 29, a text message screen is output on the screen of the mobiledevice 200 while a moving image is executed. The screen of the mobileterminal 200 is transmitted to the image display device 100 and isequally output on the image display device 100.

Although the screen of the mobile terminal 200 and the screen of theimage display device 100 are equally output in FIG. 29, differentscreens may be output according to circumstances. The mobile terminal200 determines whether the screen information is being transmitted tothe image display device 100, that is, whether a mirroring service isbeing provided, converts the screen information of the mobile terminal200 into a format suiting the connected image display device 100 if themirroring service is being provided, and transmits the converted screeninformation.

For example, as shown in FIG. 29, if a text message is received while amoving image is executed, the mobile terminal 200 may output the contentof the text message on the screen of the mobile terminal 200. However,as shown in FIG. 29, a message confirmation screen may be transmitted tothe image display device 100.

FIG. 30 shows the user inputting a first user touch to the mobileterminal 200 and FIG. 31 shows the screen of the image display device100 if the first user touch is input to the mobile terminal 200.

In FIG. 30, the user inputs the first user touch in one right lowerregion of the mobile terminal 200. Then, the user drags the first usertouch to a “YES” menu region. That is, the user inputs a touch of weakpressure in one right lower region of the mobile terminal 200 and thendrags the touch to the “YES” menu region.

In this case, the controller 280 of the mobile terminal 200 controls theoutput unit 250 to output the guide image 156 in the screen region inwhich the first user touch is input. As shown in FIG. 30, if the userinputs the first user touch in one region of the screen, the controller280 outputs the pointer in the screen region in which the first usertouch is input.

As shown in FIG. 30, if the user inputs a first user touch in one rightlower region of the mobile terminal 200 and then drags the first usertouch to the “YES” menu region, the controller 280 outputs the guideimage 156 along the region in which the first user touch is input.

Then, the controller 280 captures a current screen of the mobileterminal 200, generates screen information of the mobile terminal 200,and transmits the generated screen information to the image displaydevice 100 via the external device communication unit 271. Thetransmitted screen information of the mobile terminal 200 is informationabout the screen which the guide image according to the first user touchis further displayed.

The image display device 100 outputs the screen information of themobile terminal 200 via the display 170.

FIG. 31 shows the output of the screen information of the mobileterminal transmitted to the image display device 100, in which the guideimage 156 according to the first user touch is further displayed. Thatis, the pointer which is the guide image 156 is output while moving fromthe right lower region, in which the first user touch is input, of thescreen to the “YES” menu region.

FIG. 32 shows the user inputting the second user touch to the mobileterminal 200 and FIG. 33 shows the screen of the image display device100 if the second user touch is input to the mobile terminal 200.

In FIG. 32, the user inputs the second user touch in the “YES” menuregion of the mobile terminal 200. That is, the user inputs a touch ofstrong pressure in the “YES” menu region.

At this time, the controller 280 of the mobile terminal 200 executes theuser command corresponding to the second user touch. That is, thecontroller 280 may analyze the user command corresponding to the seconduser touch and control the mobile terminal 200 according to the usercommand.

Accordingly, the controller 280 executes the “YES” menu according to theuser command corresponding to the second user touch. The controller 280captures a current screen of the mobile terminal 200, generates screeninformation of the mobile terminal 200, and transmits the generatedscreen information to the image display device 100 via the externaldevice communication unit 271. The transmitted screen information isinformation about the screen on which the “YES” menu of the moving imageis executed.

The image display device 100 outputs the screen information of themobile terminal 200 on the display 170.

FIG. 33 shows the output of the screen information of the mobileterminal 200 transmitted to the image display device 100, in which the“YES” menu is executed. That is, as the information about the screen onwhich the “YES” menu is executed according to the second user touch, atext message is output.

In the first to third embodiments, the user touch is used in order tocontrol the screen information of the mobile terminal 200 output via theimage display device 100. That is, the user inputs the touch to themobile terminal 200 to control the screen output via the mobile terminal200 and the image display device 100.

The present invention can provide a user interface for controlling thescreen information of the mobile terminal 200 output via the imagedisplay device 100 using various input means in addition to the usertouch, which will now be described.

FIG. 34 is a diagram showing a user interface environment illustratingfourth to sixth embodiments of a user interface provision method forcontrolling the screen of an image display device 100 of the presentinvention.

The mobile terminal 200 transmits the screen information of the mobileterminal 200 to the image display device 100 through data communication.The image display device 100 outputs the screen information of themobile terminal 200 to the user.

That is, the image display device 100 receives and outputs the screeninformation of the mobile terminal 200, such that the image displaydevice 100 and the mobile terminal 200 output the same screen. Thus, theuser may receive the screen of the mobile terminal 200 via the mobileterminal 200 and the image display device 100.

In FIG. 34, the moving-image screen which is the screen of the mobileterminal 200 is transmitted to the image display device 100 and isequally output on the image display device 100.

FIG. 35 shows the fourth embodiment of the user interface provisionmethod for controlling the screen of the image display device 100 of thepresent invention. In FIG. 35, the user inputs voice “menu” to the userinput unit 272 of the mobile terminal 200.

If the user voice is input to the mobile terminal 200, the controller280 of the mobile terminal 200 executes the user command correspondingto the user voice. That is, the controller 280 analyzes the user commandcorresponding to the input user voice and controls the mobile terminal200 according to the user command.

As shown in FIG. 35, if the user voice “menu” is input to the mobileterminal 200, the controller 280 may recognize the user voice “menu” asthe user command for executing moving-image navigation. The controller280 executes the user command corresponding to the user voice. Thecontroller 280 controls the output unit 250 to output a moving-imagenavigation screen.

If the user command corresponding to the user voice is executed, thecontroller 280 enters a voice recognition mode and waits for user voicerecognition. For example, as shown in FIG. 35, if moving-imagenavigation is executed according to the user voice “menu”, thecontroller may enter the voice recognition mode in order to recognizethe user voice for controlling the navigation. This allows the user tocontinuously voice-control the mobile terminal 200. The controller 280may output the screen corresponding to the voice standby mode throughthe output unit 250 in order to notify the user of entrance into thevoice standby mode.

The controller 280 may previously set a voice command set subjected tovoice recognition after entering the voice standby mode. For example, ifmoving-image navigation is performed, the controller 280 may previouslyset a voice command set of “rewind”, “fast forward”, “play” and “stop”corresponding to moving-image navigation functions. In addition, if anInternet browser is executed, the controller 280 may previously set avoice command set of “selection”, “click” and “back” corresponding tothe functions of the Internet browser, such that the controller 280recognizes only predetermined voice commands.

The controller 280 captures a current screen of the mobile terminal 200,generates screen information of the mobile terminal 200, and transmitsthe generated screen information to the image display device 100 via theexternal device communication unit 271. The transmitted screeninformation of the mobile terminal 200 is information about the screenon which the moving-image navigation executed according to the voicecommand of the user is further displayed. The image display device 100outputs the screen information of the mobile terminal 200 via thedisplay 170.

FIG. 36 shows a fifth embodiment of a user interface provision methodfor controlling the screen of an image display device 100 of the presentinvention.

In FIG. 36, the user inputs a predetermined gesture to the user inputunit 272 of the mobile terminal 200. Although FIG. 36 shows the case inwhich the user inputs a gesture using a touch, the user may input agesture using motion via the camera 221.

If the user gesture is input to the mobile terminal 200, the controller280 of the mobile terminal 200 executes the user command correspondingto the user gesture. That is, the controller 280 analyzes the usercommand corresponding to the input user gesture and controls the mobileterminal 200 according to the user command.

As shown in FIG. 36, if an “M” gesture is input to the mobile terminal200, the controller 280 may recognize the “M” gesture as the usercommand for executing moving-image navigation. The controller 280executes the user command corresponding to the user gesture. Thecontroller 280 controls the output 250 to output the moving-imagenavigation screen.

The controller 280 captures a current screen of the mobile terminal 200,generates screen information of the mobile terminal 200, and transmitsthe generated screen information to the image display device 100 via theexternal device communication unit 271. The transmitted screeninformation of the mobile terminal 200 is information about the screenon which the moving-image navigation executed according to the voicecommand of the user is further displayed. The image display device 100outputs the screen image of the mobile terminal 200 via the display 170.

FIG. 37 is a diagram showing a sixth embodiment of a user interfaceprovision method for controlling the screen of an image display device100 of the present invention.

In FIG. 37, the user inputs a predetermined gesture to the image displaydevice 100 via the user input unit 155.

If the user gesture is input to the image display device 100, thecontroller 180 of the image display device 100 analyzes the user commandcorresponding to the input user gesture and generates user command data.The controller 180 transmits the generated user command data to themobile terminal 200 via the interface 150.

The controller 280 of the mobile terminal 200 controls the mobileterminal 200 according to the received user command data.

As shown in FIG. 37, if a gesture of putting one hand up is input to theimage display device 100, the controller 180 of the image display device100 may recognize the gesture as a user command for executingmoving-image navigation. The controller 180 generates and transmits usercommand data corresponding to the user gesture to the mobile terminal200.

The controller 280 of the mobile terminal 200 may control the outputunit 250 according to the received user command data to output themoving-image navigation screen.

The controller 280 captures a current screen of the mobile terminal 200,generates screen information of the mobile terminal 200, and transmitsthe generated screen information to the image display device 100 via theexternal device communication unit 271. The transmitted screeninformation of the mobile terminal 200 is information about the screenon which the moving-image navigation executed according to the gesturecommand of the user is further displayed. The image display device 100outputs the screen image of the mobile terminal 200 via the display 170.

According to the above-described user interface, the present inventioncan provide various input means for controlling the screen of the imagedisplay device 100. Accordingly, the user can easily control the screenshared between the mobile terminal 200 and the image display device 100.

FIG. 38 is a diagram showing a seventh embodiment of a user interfaceprovision method for controlling the screen of an image display device100 of the present invention.

As described above, the mobile terminal 200 and the image display device100 output the same screen to the user. Accordingly, the screen of themobile terminal 200 and the screen of the image display device 100,which are viewed by the user, should be the same.

However, the user views the screen of the image display device 100 in afixed direction, but views the screen of the mobile terminal 200 in adirection which may vary according to use form.

As shown in FIG. 38, although the user views the screen of the imagedisplay device 100 in the landscape mode, the user may view the screenof the mobile terminal 200 in the portrait mode or the landscape mode.

That is, if the mobile terminal 200 is in the landscape mode, the userviews the screen of the mobile terminal 200 in the landscape mode and,if the mobile terminal 200 is in the portrait mode, the user views thescreen of the mobile terminal 200 in the portrait mode.

Accordingly, the present invention provides the following user interfacein order to equally provide the screen of the mobile terminal 200 andthe screen of the image display device 100 to the user.

FIG. 38A shows the mobile terminal 200 in the portrait mode. In thisinstance, the controller 280 of the mobile terminal 200 sets right andleft margins in the screen information and transmits the screeninformation with right and left margins to the image display device 100.

As shown in FIG. 38, the controller 280 of the mobile terminal 200 mayprocess the right and left margins of the screen information in blackand white such that the screen of the portrait mode suits the screen ofthe image display device 100 of the landscape mode, and transmit thescreen information to the image display device 100.

The image display device 100 outputs the screen information of themobile terminal 200 with right and left margins via the display 170.

FIG. 38B shows the mobile terminal 200 in the landscape mode. Thecontroller 280 of the mobile terminal 200 transmits the screeninformation to the image display device 100 without change, because thescreen of the mobile terminal and the screen of the image display device100 are located in the same direction. The image display device 100outputs the screen information of the mobile terminal 200 via thedisplay 170.

If the image display device 100 is in the portrait mode and the mobileterminal 200 is in the landscape mode, the controller 280 of the mobileterminal 200 may process upper and lower margins of the screeninformation in black and white, and transmit the screen information withupper and lower margins to the image display device 100.

In the present invention, the margins may be processed using variousmethods in addition to a black and white processing method. For example,the controller 280 of the mobile terminal 200 may automatically changethe screen information of the mobile terminal 200 into a portrait orlandscape mode and may set the margins. The controller 280 mayautomatically change the screen information of the mobile terminal 200to the screen posture direction of the image display device 100 so as toremove the margins.

The controller 280 may output a hidden screen of the mobile terminal 200in the margins to fill the margins. That is, the controller 280 mayoutput the screen for displaying functions which are executed but arenot output, such as a widget, a clock, a message, etc. of a wallpaper inthe margins.

That is, the controller 280 of the mobile terminal 200 may compare thescreen posture directions of the image display device 100 and the mobileterminal 200 and determine whether margins are set in the screeninformation.

The controller 280 of the mobile terminal 200 may measure the screenposture direction of the mobile terminal 200 using a gyroscope sensor,etc. The controller 180 of the image display device 100 may measure thescreen posture direction of the image display device 100 using agyroscope sensor, etc. The mobile terminal 200 and the image displaydevice 100 may share the screen information of the mobile terminal 200and the image display device 100 through data communication.

At this time, the controller 280 of the mobile terminal 200 sets marginsin the screen information of the mobile terminal 200 and transmits thescreen information with the margins to the image display device 100 ifthe screen posture direction of the mobile terminal 200 and the screenposture direction of the image display device 100 are different.

According to the present invention, it is possible to efficientlydetermine a target mobile terminal to be controlled by a user usinglocation information of screens output on an image display device andlocation information of a cursor. In addition, if the user changes thetarget mobile terminal, the image display device can rapidly establishdata communication with the changed target mobile terminal.

In control of the screens output on the image display device using themobile terminals, the user may manipulate the mobile terminals in astate of fixing their eyes on the image display device, therebyincreasing user convenience. That is, it is possible to solveinconvenience that the user should manipulate the mobile terminals whilealternately viewing the mobile terminal and the image display device.

It is possible to provide various input means for controlling thescreens output on the image display device and allow the user to easilycontrol the screens shared between the mobile terminals and the imagedisplay device.

In addition, in display of a guide image corresponding to a user commandon the screen, a complicated protocol for calculating the input locationof the user command does not need to be mounted between devices and thuscomplexity of a user interface can be significantly reduced.

It will be apparent to those skilled in the art that variousmodifications and variations can be made in the present inventionwithout departing from the spirit or scope of the inventions. Thus, itis intended that the present invention covers the modifications andvariations of this invention provided they come within the scope of theappended claims and their equivalents.

1. (canceled)
 2. A digital television for sharing multiple data with amobile terminal, the digital television comprising: a display; a tunerconfigured to receive a broadcast signal including video signal andaudio signal; a wireless network interface configured to transmit datato the mobile terminal and to receive audio information and screeninformation from the mobile terminal wirelessly connected to the digitaltelevision; and a controller configured to: control a signal processorto decode the video signal and the screen information, control thedisplay to display first image corresponding to the decoded video signalwith second image corresponding to the decoded screen information,wherein the second image corresponds to an image displayed on the mobileterminal, and receive at least one command from a remote controller,wherein the controller is further configured to control the signalprocessor to process the audio signal and the audio information and tooutput the processed audio signal or the processed audio information toan audio output unit, and wherein the controller is further configuredto control the signal processor to change the second image's size orposition depending on the at least one command received from the remotecontroller.
 3. The digital television of claim 2, wherein the controlleris further configured to control the wireless network interface totransmit the at least one command received from the remote controller tothe mobile terminal depending on a cursor's coordinate information. 4.The digital television of claim 2, wherein the second image is displayedwith the first image in PIP (Picture-In-Picture) or POP(Picture-Out-Picture) form.
 5. The digital television of claim 2,wherein the wireless network interface is further configured to transmita decoding method supported by the digital television in response toreceiving a request from the mobile terminal.
 6. The digital televisionof claim 2, wherein the audio information and the screen information areseparately received from the mobile terminal or multiplexed informationincluding the audio information and the screen information are receivedfrom the mobile terminal.
 7. The digital television of claim 2, whereinthe wireless network interface performs based on Wi-Fi, Bluetooth orZigbee.
 8. The digital television of claim 2, wherein a first wirelesspath between the digital television and the remote controller isdifferent from a second wireless path between the mobile terminal andthe digital television.
 9. The digital television of claim 2, whereinthe controller is further configured to control the signal processor tochange the first image's size according to a change of the second image'size, and change the first image's position according to a change of thesecond image' position.
 10. The digital television of claim 2, whereinthe data includes information on a screen posture direction of thedisplay.
 11. The digital television of claim 2, wherein the second imageis an image representing an execution screen of an application installedin the mobile terminal.
 12. A method for sharing multiple data with amobile terminal in a digital television, the method comprising:receiving a broadcast signal including video data and first audio data;transmitting data to the mobile terminal; receiving second audio dataand screen information from the mobile terminal wirelessly connected tothe digital television; decoding the screen information and the videodata; displaying, a display of the digital television, a first imagebased on the screen information with a second image based on the videodata, wherein the first image corresponds to an image displayed on themobile terminal; outputting the first audio data included in thebroadcast signal or the second audio data received from the mobileterminal to an audio output unit; receiving at least one command from aremote controller; and changing the first image's size or positiondepending on the at least one command received from the remotecontroller.
 13. The method of claim 12, further comprising: transmittingthe at least one command received from the wireless remote controller tothe mobile terminal depending on a cursor's coordinate information. 14.The method of claim 12, wherein the first image and the second image aredisplayed based on PIP (Picture-In-Picture).
 15. The method of claim 12,further comprising: transmitting a decoding method supported by thedigital television in response to receiving a request from the mobileterminal.
 16. The method of claim 12, wherein the second audio data andthe screen information are separately received from the mobile terminalor multiplexed information including the second audio data and thescreen information are received from the mobile terminal.
 17. The methodof claim 12, wherein the wireless network interface performs based onWi-Fi, Bluetooth or Zigbee.
 18. The method of claim 12, wherein a firstwireless path between the television and the wireless remote controlleris different from a second wireless path between the mobile terminal andthe television.
 19. The method of claim of claim 12, wherein the methodfurther comprises: changing the first image's size according to a changeof the second image' size, and changing the first image's positionaccording to a change of the second image' position.
 20. The method ofclaim 12, wherein the data includes information on a screen posturedirection of the display.
 21. The method of claim 12, wherein the secondimage is an image representing an execution screen of an applicationinstalled in the mobile terminal.